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(57) Abstract 



A digital telemedicine imaging system (10) for enabling a clini- 
cian at a local site to observe, analyze and provide diagnostic opinions 
on medical specimens from paUents at one or more remote surgical 
sites. The system (10) includes a server system (12) at cadi remote 
surgical site and a client system (14) at the local site. The server sys- 
tems (12) are programmed with artificial intelligence-based software 
to enable the automatic identification, acquisition and bansmission of 
image regions of likely potential interest to the clinician before the 
clinician begins a diagnostic session or while the clinician is observing 
previously transmitted images. The image idendficadon. mechanical 
prepositioning and pretransmission function enhances the overall effi- 
ciency of both the imaging system and the diagnostic process supported 
by the system. 
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DIGITAL TELEPATHOLOGY IMAGING SYSTEM 
PROGRAMMED FOR IDENTIFYING IMAGE REGIONS OF 
POTENTIAL INTEREST AND ANTICIPATING SEQUENCE OF 
IMAGE ACQUISITION 

BACKGROU\P OF THE INVENTION 
Field of the Invention 

The present invention relates generally to microscope and other imaging systems 
used to analyze biological tissue specimens during diagnostic sessions. In particular, the 
present invention is a digital tdepathology imaging system incorporating anifidal 
intelligence to automatically identify tissue regions of possible interest to a pathologist, 
anticipate the sequence of image acquiation and transmit the acquired images. 

Pcscrintion of the Related Art 

Pathologists routinely assist surgeons by analyzing and providing (^agnostic 
opinions on fix)zen sections of tissue samples removed from patients during surgery. 
Frozen sections are small tissue specimens which are quick-frozsen, sliced, stained with 
dyes and placed on slides. The slides are then analyzed through a microscope by the 
pathologists. The diagnoses provided by the pathologists are used by the surgeons during 
the course of the surgery for determining the nature and extent of further surgical 
procedures. For example, pathologists often analyze tissue samples to detennfiine whether 
they are cancerous, and to determine the type of cancer present. Using this information 
the surgeon will determine which dssue should be removed from the patient. 

A typical tuntaround tune for preparing a frozen section is about fifteen minutes. 
During the tissue sample diagnosis session a pathologist will typically first rapidly scan 
the entire sample for gross features at a relatively low magnification, and identify 
particular sample regions of interest for more detailed study. The selected regions of 
interest are then observed at relatively high magnifications. It is typical for a pathologist 
to observe many separate regions at a number of different magnifications during an 
analysis session. The length of time required by the pathologist to complete this 
procedure and provide a diagnosis often is about five to seven minutes. 
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Large hospitals and universities often maintain an on-site clinical pathology 
laboratory and staff of pathologists to provide these services on demand. Since the 
diagnoses can be used to g^ide procedures to completion during a single surgery, the 
immediate access to the pathologist's expenise during surgery can provide sgnificant 
benefits in terms of both economic and recuperative aspects of healthcare delivery. 

Smaller and outlying hospitals often rely on periodic visits from ''circuit-riding*' 
pathologists that provide scheduled service to a group of hospitals. This commuting 
arrangement can be inefficient, since certain types of surgeries must be scheduled to 
coincide with the pathologist's visit. Travel is also an inefficient use of the pathologist's 
time. Furthermore, if routine or emergency surgical procedures bdng performed when the 
pathologist is not scheduled suddenly require a pathologist's expertise, the tissue sample 
must be shipped to a remote laboratory for diagnosis. The patient may then have to 
undergo surgeiy yet another time after the pathological diagnosis has been completed. 
These procedures can result in increased costs and reduced effectiveness of healthcare 
delivery. 

A known sohition to the problems associated with the lack of timely on-site 
access to the expertise of pathologists and other medical personnel is the application of 
telecommunications technology, also known as telemedicine. Telepathology imaging 
systems, a type of remotely-controlled imaging systems which use communication 
technology to enable the remote observation of medical specimens, are known. So-called 
static telepathology systems make use of a trained pathologist and a laboratoo' having a 
microscope with digital electronic imaging capabilities. Systems of this type are disclosed 
generally in the Weinstein et al. article Telepathology: Long-Distance Diagnosis, 
American Journal of Clinical Pathology, Vol. 91, pp. S39 - S42, 1989. A clinician or 
other "sender" prepares the frozen sample tissue specimen, analyzes the specimen through 
the microscope, and identifies a number of specimen regions that appear to be of interest. 
Digital images of the selected regions of interest are then generated, and electronically 
transferred to another site for analysis and diagnosis by a pathologist. The pathologist can 
also communicate with the sender (e g., by telephone) during these procedures, and 
request images of specific sample regions. Unfortunately, these static systems can be 
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inefficient due to the relatively large time delays inherent in the procedure. The accuracy 
of diagnoses rendered through the use of these systems also may be lower than those 
rendered with dynamic systems since the systems do not provide the pathologist with the 
capability of selecting the images he or she may desire. 

Dynamic systems such as those disclosed in the Weinstein et al. U.S. Patent 
5,216,596 make use of a trained clinidan and a laboratory having a microscope with a 
conventional analog video camera. ''Live" images of the specimen sample produced by 
the video camera are transmitted to a television monitor using conventional NTSC or 
PAL analog video signal protocols, of compressed and transmitted using well-known 
digital video conferencing technologies for transmission to remote sites. Although 
systems of this type offer real-time imaging capabilities, the images are often relatively 
poor in quality and have poor control-response. 

A digital telepathology imaging system which offers advantages of both "static" 
and "dynamic" architectures is discldsed in the commonly assigned United States Patent 
Application Serial No. 08/926,699» entitled Digital Telepathology Imaging System. 
Briefly, this system includes a number of "server** ^tems which are interfaced to a 
"client" system over a digital telecommunication channel. The dient system is located at a 
hospital or other local site of a pathologist, and includes a workstation, monitor and a 
keyboard or other operator-actuated user input. Each server system will typically be 
located at a hospital or other remote surgical site which does not have a staff pathologist, 
and includes a workstation, monitor, nucroscope station and keyboard or other user input. 
During airgical procedures being performed at the remote surgical site of a server system, 
a technician or other clinician will prepare slides of frozen sample or other tissue 
specimens and position the specimen slides on the microscope station. A pathologist or 
other clinician at the local ^te of the client system remotely controls the operation of the 
server system through the use of a graphical user interface (GUI) displayed on the 
monitor and the operator-actuated user input. In response to digital server system control 
commands received from the client system over the communication channel, digital still 
images of the specimen are generated by the microscope statioa transmitted to the client 
system and displayed in diagnostic-quality on the monitor. The pathologist can, for 
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example, observe an image of the entire specimen at a relatively low magnification, and 
identify panicular specimen regions for more detailed analysis at greater magnification 
levels. Server system control commands such as specimen region commands designating 
the sdeaed specimen regions and magnification commands designating the desired 
magnification are then generated by the diem system and transmitted to the server system 
over the communication channel. The server system then processes the control 
commands, manipulates the microscope stadon, and transmits to the client system 
specimen image data for the selected specimen region at the desired magnificanon. The 
communication channel will typically be a high bandwidth public network communication 
link capable of supporting packet-switched data communications. 

in practice, the transnussion times of high-resolution digital pathology images of 
the type produced by the system described immediately above can be rdatively long. This 
is especially the case when relatively low-bandwidth telecommunications links are used. 
Additional delays in the acquisition of images selected by the pathologist are due to the 
relatively slow response of the mechanical systems of the microscope station (e.g., the 
stage drive used to reposition the slide and the objective drive used to change 
magnification levels). It is therefore evident that there is a need (or improved digital 
telepaihology and other such imaging systems which avoid these delays. In particular, 
there is a need for a system capable of more quickly generating the display of specimen 
images of interest to the pathologist. 

BRCEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a diagrammatic illustration of a telepathology imaging system including 
an automatic image identification and acquisition system in accordance with the present 
invention. 

Figure 2 is a detailed diagrammatic illustration of one of the server systems shown 
in Figure I . 

Figure 3 is an illustration of the graphic user interface (GUI) menu generated and 
displayed by the client system shown in Figure 1 . 

Figure 4 is an illustration of a GUI Open Session menu. 
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Figure 5 is an illustration of a tiled slide ovemew specimen image. 

Figure 6 is an illustration of several cascaded specimen images with the GUI 
microscope control menu and zoom fector menu on the top image. 

Figure 7 is an illustration of several cascaded specimen images with the GUI 
image region selection reticle on the top image. 

Figure 8 is a flow diagram of an automatic image acquisition and transmission 
procedure in accordance with the present invention. 

Figure 9 is an illustration of a tiled overview specimen image divided into 
subiniages for evaluation in accordance with the present invention. 

Figure 10 is an illustration of a specimen region image with identified regions of 
likely potential interest highlighted by boxes surrounding the regions. 

Figure 1 1 is a flow diagram of a subimage evaluation procedure which can be 
performed in the automatic image acquisition and transmission procedure shown in Figure 
S. 
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DETAILED DESCRIPTION OF TRE PREFERRED EMBODIMENTS 
System Atxhitectureand Overview 
Figure 1 is an illustration of a 'Sirtual microscope" imaging system 10 for 
telepathology. Imaging system 10 includes an artificial intelligence-based system in 
accordance with the present invention for anticipating the sequence of image acquisition 
desired by a pathologist. In effect, this automatic image identification and acquisition 
system provides a "prediagnosis** function by identiiying ''suspicious" areas of the image 
that the pathologist is like^ to examine in greater detail, and predicting and generating 
additional images, typically at a greater magnification, of likely interest to the pathologist. 
This system offers a number of important advantages. They include the ability to acquire 
and transmit images of likely possible interest before they are specifically requested by the 
pathologist. The microscope imaging system can be mechanically repositioned on the 
basis of specific behavior patterns of the pathologist. EfiBdency of the imaging system 10 
is therd)y increased, and the latency of diagnoses reduced. 

In the embodiment shown, system 1 0 includes one or more (three are shown) 
"seiver" systems 1 2 which are interfaced to a "client" system 1 4 over a digital 
telecommunication channel 16. Client system 14 is located at a hospital or other local site 
of a pathologist, and includes a workstation 1 8, monitor 20 and operator-actuated user 
input 21 such as keyboard 22 and/or mouse 24. Each server system 12 will typically be 
located at a hospital or other remote surgical site which does not have a staff pathologist, 
and includes a woiicstation 26, monitor 28, microscope station 30 and keyboard or other 
user input (not shown). During surgical procedures being perfomied at the remote site of 
a server system 12, a technician or other clinician will prepare slides of fi'ozen sample or 
other tissue specimens and position the specimen slides on microscope station 30. A 
pathologist or other clinician at the local ate of dient system 14 remotdy controls the 
operation of the server system 12 through the use a graphical user tnter&ce (GUI) 
displayed on monitor 20 and the operator-actuated user input 21 . In response to digital 
server system control commands received fi-om the dient system 14 over the 
communication channd 16, digital still images of the specimen are generated by the 
microscope station 30, transmitted to the client system and displayed in diagnostic-quality 
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form on the monitor 20. The pathologist can, for example, observe an image of the entire 
specimen at a relatively low magnification, and identify particular specimen regions for 
more detailed analysis at greater magnification levels. Server system control commands 
such as specimen region commands designating the selected specimen r^ons, 
magnification commands designating the desired magnification and illumination 
commands for controlling specimen illumination and imaging parameters of the camera 
are then generated by the cbent system 14 and transmitted to the server system 12 over 
the conununication channel 16. Server system 12 processes the control commands, 
manipulates the specimen illumination and camera parameters, and transmits to the client 
system 14 specimen image data for the selected specimen region at the desired 
magnification. 

Communication channel 16 will typically be a hig^ bandwidth public network 
communication Unk capable of supporting packet-switched data communications. For 
example. ISDN (integrated semces digital network) or Tl telephone lines can be used. 
Conventional and commercially available digital switching or routing systems (not shown) 
can be used to intei^ice server system 1 2 and client system 1 4 to the communication 
channel 16. Prototypes of system 10 were developed using an ATM-based 
(asynchronous transfer mode) optical channel 16 and an Ethernet network. 

A representative server system 12 can be described in greater detail with reference 
to Figure 2. Microscope station 30 is a conventional and commercially available 
instrument which includes a microscope 3 1 and a stage 34 for receiving and supporting 
specimen slides (not sqaarately shown) to be imaged by lenses or objectives 36. The 
stage 34 is mounted to the base of the microscope 3 1 by an x-y stage drive 40 so the 
position of the slide with respect to the objectives 36, and therefore the region of the 
specimen being imaged, can be changed. The microscope 31 includes several objectives 
36, each of which provides a different magnification power. Focusing is peribrmed by a 
conventional automatic focus control system (not separately shown). The objectives 36 
are driven and positioned by a drive 38 to change the magnification of the microscope. 
Although not shown, the microscope 3 1 also includes a conventional slide illumination 
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light source and actuator-driven neutral density filter for controlling the specimen 
iihimination level. 

A higih-resolution camera 42 generates true-color image data representative of the 
specimen images produced by the microscope. Tlus image data is digitized by the 
workstation 26. The camera 42 and digitizing system in workstation 26 function as a 
digital imaging system. Prototypes of system 1 0 include a camera 42 having a base 
resolution of at least approximately 600 x 800 pixels, and a digitizing system capable of 
capturing 8-bit per pbcel (bpp) per color band still images (for a total of 24 bpp color). 
Conventional imaging control parameters of camera 42 can be accessed and controlled. 

Microscope station 30 is interconnected to workstation 26 through one or more 
interfaces illustrated generally at 44. Prototypes of system 10, for example, include an 
analog inter^ for connecting camera 42 to the workstation 26 and its digitizing system, 
and RS-232 busses for interconnecting the stage drive 40, objective drive 38, automatic 
focus system and image illumination system to the workstation 26. Monitor 28 is 
interconnected to workstation 26 by bus 46. These systems are capable of acquiring a still 
image and displaying that image at full resolution on the monitor 28 in less than about 1 
second. 

Workstation 26 is a conventional programmable computer, and is programmed 
with server system control software to control the operation of microscope station 30 in 
response to digital server system control commands received from the client system 14. 
Control commands received fix)m server system 1 4 include camera commands, spedmen 
region commands, magnification commands, focus commands and illumination 
commands. The specimen region commands include information designating a specific or 
desired r^on of interest on the specimen slide, and in one embodiment include 
information designating coordinates on the specimen slide in terms of "stage units." In 
response to the receipt of specimen region commands, workstation 26 actuates stage 
drive 40 and moves stage 34 to position the selected region of the specimen slide under 
objective 36. The magnification commands include information designating the desired 
magnification at which the selected specimen region is to be imaged. In response to the 
receipt of magnification commands, workstation 26 actuates drive 38 and causes the 
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selected objective 36 to be used to image the specimen region. In response to camera 
commands, camera 42 generates specimen image data representative of a still image of 
the specimen region being imaged by the microscope 3 1 . As described above, this image 
data is digitized by the workstation 26. 

As noted above, microscope 3 1 includes automatic focus and specimen slide 
illumination control systems. In response to focus commands received from client system 
14, workstation 26 initiates an automatic focus routine by the automatic focus system of 
the microscope 3 1 . Workstation 26 also initiates an automatic focus routine each time the 
magnification is changed. Workstation 26 also controls the illumination system of the 
microscope 3 1 (e.g., the neutral density filter) to change the illumination level of the 
specimen slide in response to illumination conrunands received from the client system 1 4, 
and can vary the imaging parameters of camera 42. 

Prototypes of system 10 use conventional TCP/IP protocol (transport control 
protocol/internet protocol) for the transfer of the server system control commands and 
specimen image data. The TCP/IP protocol of the prototype system 10 is supported by 
the Windows Sockets application programming interface (API) which provides 
handshaking, error reporting and associated fiincdons, and is therefore compatible with 
local area network (LAN) as well as LAN-over-broadband telecommunications 
architectures. 

Client system 14 can inchide a workstation 18 and monitor 20 similar to those of 
server system 12. Workstation 1 8 is programmed with client system control software and 
GUI software. The GUI software generates and displays a Windows'^-format GUI on 
monitor 20. Through the use of the GUI and user input 2 1 , the pathologist or other 
clinician can select the specimen regions to be imaged by the microscope station 30 of a 
server system 12, and the desired magnification of the selected specimen region. Server 
system control commands, inchiding the spedmen region commands and magnification 
commands designating the image selected by the pathologist, are generated by the client 
system control software and transmitted to the server system 12 over communication 
channel 16. The client system control software also processes the digital image data 
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received from the server system 12, and generates visual displays on monitor 20 of the 
associated specimen images represented by the data. 

Graphic User Interface And System Operation 
Figure 3 is an illustration of the GUI menu 50 generated by workstation 1 8 and 
displayed on monitor 20 at client system 14. The iihistrated embodiment of menu 50 
includes a menu bar 52 and a toolbar 54. The pathologist operates system 10 by using 
keyboard 22 and/or mouse 24 in a conventional manner to select commands on menu bar 
52 and toolbar 54. As described below, other commands not graphically displayed on the 
GUI can be selected by clicking the left and right buttons (not separately shown) of the 
mouse 24. in addition to conventional Windows™ commands (e.g., Window and Help), 
menu bar 52 inchides the system-specific commands System, Session and Microscope. 
Toolbar 54 inchjdes a number of conventional Windows™ command buttons 55 - 58, 
microscope (e.g., stage drive) control buttons 59, and a new slide button 60. Buttons 55 
and 56 are format buttons for selecting tiled or cascaded specimen image displays, 
respectively. Arrange icon button 57 and information button 58 provide conventional 
Wmdows™ fiinctions when selected. Get slide button 60 can be actuated by the 
pathologist to obtain a tiled slide overview image (described in greater detail below) when 
a new specimen slide has been positioned for imaging on the remote microscope station 
30. The tiled slide overview image can also be initiated by the technician at the server 
system 12. A listing of the commands and associated fimctions available through the 
selection of the System, Session, Microscope, Window and Help commands on menu bar 
52 follows Many of the commands provide conventional Windows™-format functions, 
and are identified as ''Standard'' in the listing. 
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Menu Bar Options 



System 



Preferences 


Display *'user piefeience" options to set paiam^rs such as the u-ay new- 
images are displav'ed. the width of reticle borders, the way that gamma 
correction is performed, and the server sites that arc available 


Save 


Standard 


Save As 


Standard 


Print 


Standard 


Print Setup 


Standard 


Exit 


Standard 


Session 


Connect 


Establish diagnostic session with server system (remote microscope) site 


Gamma 

Correction 


Perform gamma correction on an existing image accordii^ to Kttiqgs in 

"Preferences*' menu 


Get Slide 


Retrieve overview of entire specimen 


Remote 
Database 


Establish session with existing remote database (as opposed to remote control 
of a microscope) 


Chat 


Establish TCP/IP "chat" session with remote node to intcraa with remote 
user via keyboard 


Submit 
Diagnosis 


Display form for pathologist to complete with info specific to this diagnostic 
session, including fields for diagnosis, confidence level, system performance, 
image qualit>\ etc. dien send completed form to microscope location to be 
induded with 'trajectory'* and patient history. This option finalizes the 
session 


Patient 
Inibmiation 


Display summary infimnation retrieved from electronic patient record or 
entered by tedmidan at microscope site 
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Microscope 



Rcfocus 


Instruct remote microscope to perfbrni autofoctis routine (automatically 
retrieves another image) 


Intensit)' 


Instnict remote microscope to change iliumination of specimen via neutral 
density filter and/or camera parameters 


Get Image 


Retrieve another copy of the current image fiom remote microscope 


Image 
Calibrate 


Perform interactive adjustment of image brightness to remotely calibrate the 
microscope's camera settings 



Window 



Tile 


Standard 


Cascade 


Standard 


Arrange All 


Standard 


About 


Standard 



Help 



Contents 


Standard 


Search for Help 


Standard 


How to Use Help 


Standanl 



Figure 4 is an illustration of a GUI open session menu displayed on monitor 20 
and used by the pathologist to initiate a remote specimen observation sessioa The open 
session menu will be presented to the pathologist upon the selection of the Connect 
command available through menu bar 52. As shown, the open session menu includes a 
list of server systems 12 set up for operation with the client system 14, and displays the 
then-current specimen slide illumination (intensity) and image magnification (the 
established * Wr preference" values). Workstation 1 8 initiates and establishes an 
operational communication imerface with the server system 12 selected by the pathologist 
through the open session menu. 
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The pathologist will typically begin the diagnosis of a specimen by observing a 
slide oveIvie^^• image 70 such as that shown in Figure 5. The slide overview image 70 is a 
relatively low magnification image of a relatively large portion of the specimen on the 
slide, and in one embodiment is an image of the entire specimen on the slide. In the 
embodiment shown, the slide overview image 70 is an assembly of separate and discrete 
specimen region images 72 into a composite or tiled overview image of the entire 
spedmen on the slide. The slide overview image 70 shown in Figure 5, for example, is 
formed from a 3 x 4 array of specimen region images 72. 

Generation of the overview image 70 is controlled by the server system control 
software in workstation 26. In response to Get Slide commands (or Send Slide 
commands initiated by the clinician at server system 12), the workstation 26 actuates 
objective drive 38 to select a rdatively low magnification (e.g.. 2x) objective 36 and 
initiates an automatic focus routine. Stage drive 40 and camera 42 are then actuated to 
sequentially obtain the adjacent specimen region images 72. The image data for the 
specimen region images 72 is processed by the server system to reduce the resolution of 
the images, and transmitted to the client system 14 where it is assembled and displayed as 
the slide overview image 70. The image data for each of the specimen r^on images 72 
can be cached in memory of the woiicstation 1 8 so the image can be redisplayed as the 
topmost image on monitor 20. 

As noted above, the generation of a slide overview image 70 can be initiated at 
either the server system 12 or the client system 14. After positioning a new specimen 
slide on nucroscope station 30, the clinician can, for example, initiate the generation of a 
slide overview image 70 by selecting a Send Slide command through a graphic user 
interface and user input (not separately shown) at the server system 12. Image data 
representative of the slide overview image 70 wiU then be transmitted to the client system 
14 and displayed on monitor 20, thereby indicating to the pathologist the availability of 
the specimen slide for observation. If the image data has already been transmitted by the 
server system 12, the pathologist can also initiate the generation of a new slide overview 
image 70 from the client system through use of the Get Slide command button 60 on 
toolbar 54 or the Get Slide conunand available through menu bar S2. 
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During observation of the slide overview image 70 the pathologist will typically 
identify specimen regions that he or she would like to observe in greater detail Using the 
GUI and user input 21, the pathologist can select the specimen region of interest and the 
desired magnification at which the specimen region is to be observed. The appropriate 
server system control commands are then generated by the client system control software 
and transmitted to the server system 1 2 to initiate the retrieval and display of the specimen 
region of interest image (ht., a detail image). 

Specimen re^ons of interest can. for example, be selected by positioning the GUI 
cursor over the image and clicking the right button on mouse 24. As shown in Figure 6, 
this command causes a GUI microscope control menu 80 to be generated and displayed 
on monitor 20. Available commands in the microscope control menu 80 include Zoom 
and Zoom Factor When the Zoom command is selected through user input 21, the GUI 
software generates and displays on the then-displayed specimen region image an indicator 
such as reticle 82 shown in Figure 7. Using the user input 2 1 the pathologist can then 
reposition the reticle 82 over the specimen region of interest on the image, and thereby 
sdea the specimen region for ftirther observation. Returning to the microscope control 
menu 80, the pathologist can then select the Zoom Factor command and cause the zoom 
factor menu 84 (Figure 6) to be generated and displayed by the GUI software. The zoom 
factor menu 84 includes a number of discrete magnification level commands 
corresponding to the available magnification levels (e.g., objectives 36) on microscope 
station 30, as well as In and Out step commands. The In command causes the 
magnification to increase one available power level fi-om the then-cunrent magnification at 
the microscope station 30, while the Out command causes the magnification to decrease 
one available power level ft^om the then-current magnificatioa By clicking the left button 
of mouse 24 after sdeaing the desired magnification and image region of interest in the 
manner described above, the pathologist will initiate the generation and transniission of 
the appropriate magnification and specimen region commands to retrieve the image. 

Using the client system 14 in the manner described above, the pathologist is able 
to relatively quickly observe images of many different specimen regions at different 
magnifications. As is evident from Figure 6, multiple images can be simultaneously 
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displayed on monitor 20. The images can be displayed in a manner w-hich enables the 
pathologist to easily identify the location of the spedmen region image with respect to the 
image overview and with respect to all other images. The magnification of the image is 
also displayed. Workstation 26 automaticaUy maintains a log or cache describing in 
chronological sequence the control commands transmitted to the server system 12 and the 
coordinates of the specimen region of the retrieved image. In addition to providing an 
audit trail, this infoimation streamlines the image acquisition and transfer functions and 
provides the pathologist with immediate feedback regarding the regions of the specimen 
which have been imaged. Each of the retrieved specimen images can be stored by the 
workstation 26 for subsequent retrieval and observation. 

System 10 offers a number of important advantages, especially with respect to 
known "static'* and ^'dynamic" architectures. High-resolution, diagnostic-quality, discrete 
color images can be retrieved and displayed in real time. The system can be efficiently 
operated through the user-friendly graphic user interface. Commercially available 
hardware can be used to implement the system. The network protocol is also robust, 
scalable (works on many different size channels), fast and relatively secure. The system 
allows pathologists to quickly provide high-quality diagnostic opinions to multiple remote 
locations, thereby making efficient use of their time. Accurate, precise, real-time control 
of the remote imaging system is achieved. The system can be integrated with electronic 
patient record systems to include patient information and diagnostic results. The audit 
trails which can be provided are also advantageous. 

Automatic Image Identification and Acquisition Function 

Artifidal intelligence-based automatic image identification and acquisition 
software \n accordance with the present invention is incorporated into the server system 
control software of workstation 26. The image identification and acquisition software 
identifies locations on previously-generated images such as 70 diat will likely be of 
potential interest to the pathologist (e.g., are "suspicious"). The image identification and 
acquisition software also predicts which additional images (e.g., location and 
magnification) are likely to be of potential interest to the pathologist. These fiinctions of 
the image identification and acquisition software are typically performed in the absence of 
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express commands from the pathologist or client system 14, and can take place while 
previously selected image data is being transmitted to the client system. On the basis of 
these computer-assisted decisions the server system control software can control the 
operation of workstation 26 and/or microscope station 30 to effectively provide 
"prediagnoses" and enhance the overall efficiency of both the imaging system 10 and the 
diagnostic process supported by the imaging system. 

The automatic image identification function can be described generally with 
reference to the flowchart in Figure 8. As shown at step 200, the automatic image 
identification fijnction is initiated by generating a composite or tiled overview image. 
Figure 9 is an illustration of a composite or tiled overview image 100 of a stained tissue 
specimen 102 mounted on a slide (not shov^). Overview image 100 is a composite 4x2 
array of relatively low magnification (e.g., 2x) specimen region images 104. Following 
the generation of the overview image 100, and in some embodiments while the overview 
image is being transmitted to the client system 14, the image identification and acquisition 
sofhvare divides or breaks up the overview image into a plurality of subimages 106 (step 
202). The subimages 106 will typically contain a relatively small number of pbcels, and in 
one embodiment have about IK to 4K pixels. In the embodiment described with 
reference to Figure 9, the image identification and acquisition software divides the 
overview image 100 into subimages 106 formed by a rectangular array of immediately- 
adjacent pixels in two-dimensional Euclidean space. In other embodiments (not shown) 
the grouping of neighboring pbcels in each subimage can be based on measures of 
"closeness" in other spaces (e.g., various color spaces). 

Using neural net and/or conventional or otherwise known artificial intelligence 
methodologies (e.g., image processing techniques), the subimages are evaluated to 
identify regions of likely potential interest to the pathologist (step 204). The evaluation 
performed at step 204 uses a stored tissue-specific training set of infisrmation. The tissue- 
specific training set contains information which describes charaaeristics and features of 
the specimen image that are likely to be associated with portions of the image that are of 
likely potential interest to the pathologist. The tissue-specific training set of information 
can include information which varies fi^om pathologist to pathologist (i.e., is often at least 
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panially patholoQSt-spediic) since different pathologists will look for difierent 
characteristics and features to identify areas of potential interest. For example, color-blind 
pathologists will look for different characteristics than those with good color vision. The 
subimages are effectively evaluated at step 204 to determine whether they contain image 
content similar to that characterized by the tissue-specific training set. 

For example, in frozen sections stained with conventional pink and blue dyes, it is 
known that relatively blue portions of the image are most likely to be cancerous and 
malignant, and therefore of likely potential interest to the pathologist. Fuithermore, the 
greater the amount of blue color in a portion of an image, the more likely the portion of 
the image will be of interest to the pathologist. Conversely, the more pink a portion of an 
image is, the less likely it is to contain cancerous cells. The color of the portion of the 
tissue specimen being evaluated, also known as the color-space, is therefore a criterion 
that can be incorporated into the tissue-specific training set. 

Another characteristic of tissue samples having a known relationship to regions of 
potential interest is the degree of homogeneity of the regions, also known as the spatial 
frequency-space. Healthy tissue will generally have cell nuclei which appear as small, 
isolated spots inside a regularly shaped (e.g., elliptical or circular) membrane. The bulk of 
the cell contains relatively large sections of matter of relatively consistent or unvarying 
nature (e.g., relatively large patches of pink-stained matter. Cancerous tissue, on the 
other hand, typically contains cells fbmned to a large degree by cell nuclei, and each 
nucleus has a well defined mass of DNA. Cancerous tissue therefore tends to be "busier " 
having a higher degree of variation. Known mathematical algorithms (such as those for 
performing discrete cosine transforms) can be used to evaluate the spatial frequency of 
subregions. The spatial frequency of the portion of the tissue specimen under evaluation 
is therefore another criterion that can be incorporated into the tissue-specific training set. 

The color-space and frequency-space tissue specific training set information 
described above is only an example of the t>pes of such information which can be used. 
Other types of tissue-spedfic training set information (e.g.. templates of organ-specific 
anomalies) can be used as alternatives or in addition to those described above. 
Furthermore, the particular color-space and frequency-space training set information 
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described above is directed to specific types of cancers and the conunonly used pink and 
blue dye sets. This infonnation will vary depending upon factors such as the type of 
cancers or other tissue material being looked for, the specific type of organs from which 
the specimen was removed, and the colors of the dye sets used to stain the specimen. 

The tissue-specific training sets of infomiation can be generated in a number of 
different manners. For example, they can be programmed on the basis of observations of 
a skilled pathologist (i.e., empirically identifying the characteristics of specimen r^ons 
selected by a pathologist as being of interest). Altemativdy, a neural net can be taught the 
tissue-specific training set in a conventional manner through use of electronic results of 
evaluations perfomied by pathologists (e.g., the stored caches of commands from 
previous diagnostic sessions on similar tissues with similar stains for similar diseases). 

AR& regions likely potential interest have been identified, they are ranked as 
shown at step 206. Ranking step 206 is performed to assign a relauve degree of interest 
or priority to the identified regions. Factors and methods for performing the ranking 
operation are described below. 

After they have been ranked, the image identification function continues by taking 
aaion on the identified regions as indicated generally by step 208. The identified regions 
will typically be acted upon at step 208 in order of the ranking assigned at step 206. As 
shown in Figure 8, a number of different actions can be initiated by the image 
identification and acquisition program at step 208. For example, the identified areas of 
likely potential interest can be highlighted on the tiled overview image 100 (step 210). 
Figure 10, for example, illustrates an image of a tissue sample in which identified areas of 
potential interest have been highlighted by surrounding them with bounding boxes. 
Although not visible in Figure 10, a feature of the box such as its color or line width can 
be used to indicate the ranking assigned to the identified region at step 206 (e.g., red 
boxes around the highest ranked regions of interest, and yellow boxes around r^ons of 
interest having a lower ranking). The highlighting fiinction can be performed relatively 
quickly. Since the overview image will typically have been previously transmitted to the 
client system 14, the server system 12 needs only to transmit the coordinates on the 
overview image at which the highlights should appear. By observing an overview image 
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highlighted in this manner, the pathologisi is able to quickly identify regions that he or she 
would like to view in greater detail, thereby enhancing the eflfidency of the diagnostic 
process. 

Additional and/or alternative actions which can be initiated by the image 
identification and acquisition program include the acquisition and transmission of 
additional images that are of likely potential interest to the pathologist (step 2 1 2). This 
action is undertaken by the server system workstation 26 which actuates the stage drive 
40 to position the identified region of the specimen 102 for imaging, actuates the objective 
drive 38 to select the desired (and typically higher magnification) objective 36, and then 
initiates an automatic focus routine. The camera 42 can then be operated to generate the 
specimen image data of the identified region of likely potential interest being imaged by 
the microscope 3 1 . This image data can be transmitted to the client system 14, where it 
will be stored for possible display and observation by the pathologist. 

The automatic image acquisition and transmission fimction described above will 
typically be perfomied on the highest ranked identified region of likely potential interest. 
The function can also be'repeated for additional identified regions of Ukely potential 
interest as indicated by flow path 2 1 4. Repetition of this type will typically be performed 
on identified regions in sequence of their ranking. 

Altematively, rather than continuing to acquire and transmit more detailed images 
of identified regions (i.e., rather than following flow path 214), a detailed image acquired 
at step 21 2 can be broken into subimages which are then evaluated and ranked to identify 
regions of likely potential interest therein (i.e., steps 202, 204 and 206 are repeated for the 
detailed image). Any desired action, including those described above, can then taken on 
these identified regions of likely potential interest in the detailed image. To evaluate 
subimagers at greater magnifications workstation 26 can scale the information in the 
training set being utilized for the particular magnification ^ctor of the subimage. These 
automatic image acquisition and transmission functions can be stopped by the pathologist 
through use of the user input 21 of the client system 14. Direct pathologist control over 
the server system 12 through use of user input 2 1 will typically take precedence and 
temporarily ovenride the automatic image acquisition and transmission fiinctions. 
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The automatic acquisition and transmission of detailed images of identified 
regions of likdy potential interest in the manner described above offers important 
advantages. Mechanical repositioning of the stage drive 40 and objective drive 38, and 
the autofocus routine, are relatively time consuming procedures. Transmission of the 
image data can also take time. If the pathologist were to initiate these activities through 
the diem system user input 2 1 following his or her observation of an image, these 
response times cause delays which reduce the efficiency of the pathologist and increase 
the latency of diagnoses. Undertaking these "prepositioning** and/or *'pretrajismission*' 
actions before the pathologist is even available for diagnoses, or while the pathologist is 
reviewing previously transmitted images, results in the detailed images that the pathologist 
is likely going to want to observe being present for effectively immediate display on the 
client system monitor 20 when requested. The pathologist's time is thereby used more 
efficiently. Latency of diagnoses is also reduced. 

Figure 1 1 is a flow diagram of one artificial intelligence method 204' which can be 
performed by workstation 26 of server system 12 to evaluate subimages and identify 
regions of likely potential interest (i.e.. step 204 of the method shown in Figyrc 8). 
Evaluation method 204' is a first order approach based on a color-space training set of 
information, and is presented as an example of an image processing-type of artificial 
intdligence evaluation. The method 204' begins with the transformation of the RGB 
(red/green/blue) color coordinates of all the subimage pixels (i.e., the image data) into a 
suitable color-space. In this example the color coordinates are transformed into the HSV 
(hue/saturation/value) space (step 220). Equivalent results can be obtained in many other 
color-spaces by varying processing steps, initial values, etc. 

Using the hue channel, which represents an index of "color," a two-sided 
thresholding operation is perfonmed around pbcd values representing the *1)lue" color 
band (i.e., the color if interest). In one embodiment this thresholding operation is 
performed around a blue color band represented by pixel values of about "184 - 200". In 
another version the blue color band is represented by pixel values in the more limited 
range of "190 - 19T*. AH pixels which are within the threshold range (i.e., are "fairly 
blue") are set to binary " I Pbcels which are not within the range and are therdbre not 
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sufficientiy blue to be of potential interest are set to a binary "0", In this manner the 
subregion image is effectively converted into a **blue and ^^ite** binaiy image (step 222). 

Conventional image processing morphological operations of erosion (step 224) 
and dilation (step 226) are performed on the binary image. The erosion operation 
elinunates relatively isolated "blue" pixels, much like median fikering in a binary space. 
The dilation operation strengthens or enlarges remaining "blue" pixels (which are not 
isolated), effectively causing "blue'' pbcels in a localized neighborhood to **grow 
together." 

Next, as shown by step 228, the blob characteristics (e.g., centroid, spread, 
perimeter, number of pbcels) of the "blue" pbcel groups are computed in a conventional 
manner to detemiine the size of the identified region of likely potential interest. These 
blob diaracteristtcs can, for example, be used to determine the size and location of the 
bounding boxes which are overlaid on the original image. 

The bounding boxes or other information describing the size and shape of the 
identified regions of likely potential interest can be used to rank or prioritize the identified 
r^ons (step 206 in Figure 8). For example, the area of the identified region (e.g., the 
number of pbcels in the bounding box) can be used as a measure of importance. Another 
characteristic which can be used as a measure of importance is the concentration of 
relatively small bounding boxes (e.g., the number of spatially co-located identified 
regions). Yet another characteristic is the compacmess or density of "blue" pbcels in each 
identified region. These characteristics can be used individually or in combinations to 
prioritize the identified regions of likely potential interest for subsequent actioa 

The image acquisition and transmission procedure described above with reference 
to step 212 in Figure 8 can make use of artificial intelligence methods to enhance the 
likelihood that the detailed images which are automatically acquired and transmitted are 
images that a pathologist will want to observe (i.e., images that the pathologist would 
have selected). An object of this procedure can be to acquire and transmit detailed images 
in a sequence or ^'trajectory" that matches the sequence the pathologist would have 
followed. To obtain this objertive, artificial intelligence techniques can be applied to the 
image acquisition and transmission procedure 212. 
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A pathologist-specific training set of information can, for example be used in the 
image acquisition and transmission procedure 212. It has been observed that the behavior 
of individual pathologists during image diagnosis sessions often exhibits specific 
tendencies. In other words, the manner by which they observe specimen images lends to 
be similar from specimen to specimen. By way of example, some pathologists scan an 
image (i.e., the sequence in which they acquire images of regions of interest) in a spiral 
pattern, while others scan images in rows like the text on this document. Some 
pathologists tend to jump or skip from the rdativdy low magnification (e.g., 2x or 4x) of 
the overview image such as 100 directly to relatively high magnification (e.g., 20x) images 
of regions of interest, while others will increase the magnification of sequential images in 
more gradual steps (e.g., 2x to 6x to 2Qx). Other infiannation that can be incorporated 
into the pathologist-specific training set includes preferred specimen illumination settings 
and camera imaging parameters settings. Furthermore, these pathologist-spedfic 
tendencies can vary with tissue-specific information such as the type of organ from which 
the specimen is taken and the suspected tumor or disease type (e.g., how does the 
pathologist observe specimens for different types of tissue), and/or with patient-spedfic 
information such as age. gender and condition. Re^gions of likely potential interest can 
also be identified as a fiinction of the length of time a pathologist spends observing images 
having specific types of features. 

As noted above, the server system workstation 26 can automatically maintain a 
log or cache describing in chronological sequence the control conunands transmitted to 
the server system 12 and the coordinates of retrieved specimen image regions. These 
stored histories, alone or in combination with the associated image data, can be used to 
develop training sets for associated pathologist-spedfic, tissue-specific and patient- 
spedfic information. Neural nets can also be taught with these stored histories. 
Alternatively, the pathologist-specific, tissue-specific and palicnt-spcdfic information for 
the training sets can be empirically derived by observing pathologists during diagnostic 
sessions associated with the type of information desired. Furthermore, pathologist- 
specific information can be obtained "on the fl/' during a g?ven diagnostic session, and 
used to control the acquisition of subsequent images. 



-22- 



wo 99/30264 



PCT/US9W23760 



The application of predictive methodologies such as those described above to the 
automatic acquisition and transmission of images of identified regions enhances the 
previously described advantages of the automatic acquisition function. In panicular, these 
methodologies increase the likelihood that the detailed images which are automatically 
acquired and transmitted will be those that the pathologist is most interested in observing, 
and present for essentially immediate display and observation. 

Although the present invention has been described with reference to preferred 
embodiments, those skilled in the art will recognize that changes can be made in form and 
detail without departing from the spirit and scope of the invention. In particular, although 
described in connection with a tdepathology system, the invention can be applied to other 
microscope or imaging systems used to observe stained or other biologic tissue samples 
(e.g.. Pap smear analyses). 
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WHATISCXAIMEDIS: 

1 . A method for operating a digital imaging system to automatically identiiy 
regions on a biologic tissue specimen of likely potential interest to a clinician providing 
diagnostic opinions on the specimen, including: 

providing a training set of tissue-specific information representative of tissue 
specimen characteristics of likely potential interest to a clinician; 

generating a first specimen image at a first magmfication; 

evaluating the first specimen image as a function of the training set of infomialion 
to identify interest regions of likely potential interest; and 

taking aaion on at least one of the identified interest r^ons. 

2. The mediod of claim 1 wherein evaluating the first spedmen image 
inchides: 

breaking the first specimen image into a plurality of subimages; and 
evaluating the subimages as a fiinction of the training set infoimation to identiiy 
the interest regions. 

3. The method of claim 1 wherein: 

the method fiirther includes displaying the first specimen image; and 
taking action on at least one of the identified interest regions includes highlighting 
at least one of the displayed identified interest regions. 

4. The method of claim 3 wherein: 

the method fiirther includes ranking the interest r^ons in terms of their relative 

potential interest; and 
highlighting the identified interest regions includes highlighting the interest regions 

as a fijnction of their ranking. 
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5. The method of claim 1 and funher including ranking the identified interest 
regions in terms of their relative potential interest 

6. The method of claim 1 wherein taking action on at least one of the 
identified interest regions includes generating a detailed specimen image of a first of the 
identified interest regions at a magnification which is greater than the first magnification. 

7. The method of claim 6 and fiirther including transmitting the detailed 
specimen image of the first identified interest region to a remote location. 

8. The method of claim 7 and fiirther including: 

generating a detailed specimen image of a second of the identified interest regions 
at a magnification which is greater than the first magnification; and 

transmitting the detailed specimen image of the second of the identified interest 
regions to the remote location. 

9. The method of claim 8 and fiirther including repeating the steps of 
generating and transmitting a detailed specimen image at a magnification greater than the 
first magnification for at least a third of the identified interest regions. 

1 0. The method of claim 9 wherein: 

the method further inchides ranking the identified interest r^ons in terms of their 

relative potential interest; and 
the steps of generating and transmitting detailed specimen images are performed 

on identified interest regions as a fiinction of their ranking. 

1 1 . The method of claim 10 wherein: 

the method fiirther includes providing a training set of clinician-specific 

information representative of a clinician's specimen observation behavior; 
and 
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generating detailed specimen images includes generating the detailed specimen 
images as a function of the training set of clinician-specific Infonnation. 

12. The method of claim 1 1 wherein: 

providing the training set of clinician-specific information includes providing a 

training set of information including information representative of a 

clinician's d^led image magnification behavior; and 
generating detailed specimen images includes generating detailed specimen images 

at magnifications detehnined as a function of the information 

representative of the image magnification behavior. 

1 3 . The method of claim 6 wherein: 

the method further includes ranking the identified interest regions in terms of their 

relative potential interest; and 
generating and a detailed specimen image includes generating a detailed specimen 

image on a relatively highly ranked identified interest r^on. 

14. The method of daim 6 wherein: 

the method further includes providing a training set of clinician-spedfic 

information representative of a clinician's specimen observation behavior; 
and 

generating detailed specimen images includes generating the detailed specimen 
images as a ilinction of the training set of clinician-specific iztfbrmation. 

1 5. The method of claim 14 wherein: 

providing the training set of clinician-specific information inchides providing a 
training set of infonnation including information representative of a 
clinician's detailed image magnification behavior, and 
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generating detailed specimen images includes generating detailed specimen images 
at magnifications determined as a function of the information 
representative of the image magnification behavior. 

1 6. The method of claim 1 5 wherein: 

providing the training set of clinician-specific information includes providing a 
training set of information including information representative of a 
clinician's interest region scanning sequence behavior, and 

generating detailed specimen images includes generating detailed specimen images 
on identified interest regions in an order determined as a fiinction of the 
information representative of the scanning sequence behavior. 

17. The method of claim 1 4 wherein: 

providing the trmning set of clinician-specific information includes providing a 
training set of information including information representative of a 
clinician's interest region scanning sequence behavior, and 

generating detailed specimen images includes generating detailed specimen images 
on identified interest regions in an order determined as a function of the 
information representative of the scanning sequence behavior. 

18. The method of daim 6 and fiirther including: 

evaluating the detailed specimen image as a function of the training set of 

information to identify interest regions of likely potential interest; and 

taking action on at least one of the identified interest regions in the detailed 
specimen image. 

1 9. The method of daim 1 8 wherein: 

the method further includes ranking the interest regions in the detailed specimen 
image in terms of their rdative potential interest; and 
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highlighting the identified interest regions includes highlighting the interest regions 
as a junction of thdr ranking. 

20. Operating a server system workstation in accordance with the method of 
claim 1 , wherein the server system workstation is configured for use in connection with a 
digital telemedicine imaging system of the type having a server system at each of at least 
one remote surgical sites and a client system at a local site, for enabling a clinician at the 
local site to observe, analyze and provide diagnostic opinions on medical specimens from 
patients at the remote surgical sites, the client system workstation adapted to be 
interconneaed to a digital communication channel for telecommunication with the server 
system. 
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